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OVERVIEW 

Human  performance  executing  search  and  rescue  type  of  navigation  is  one  area  that  can  benefit  from 
augmented  reality  technology >  when  the  proper  computer  generated  information  is  added  to  a  real  scene. 
Search  and  rescue  is  characterized  by  the  need  to  completely  inspect  a  space,  find  an  objective,  and  exit  the 
space.  Time  is  of  the  essence  in  completing  this  type  of  task  and  the  environment  is  normally  not  familiar  to 
the  user,  and  lacks  known  landmarks.  We  briefly  report  on  an  experiment  that  demonstrated  the  benefits  of 
augmented  reality >  in  a  search  and  rescue  task.  Specifically,  120  participants,  equally  divided  by  gender,  were 
tested  in  speed  and  accuracy  using  augmented  reality  in  a  search  and  rescue  task.  Accuracy  performance 
was  improved  using  augmented  reality  as  compared  to  the  control  conditions.  Additionally,  a  user  controlled 
On-Demand  display  resulted  in  better  performance  than  a  Continuously  On  display. 

We  report  on  additional  analysis  performed  on  data  gathered  during  the  augmented  reality >  experiment. 
Specifically  the  data  involves  whether  the  sense  of  presence  occurred  during  the  task,  how  one  can  measure 
presence  in  augmented  reality  environments,  and  the  extent  to  which  individual  differences  were  factors  in 
performance.  While  presence  was  not  found  in  the  experiments,  new  insights  and  measurement  approaches 
emerged.  Individual  differences  also  exhibited  some  interesting  results.  Spatial  recall  and  ingress-egress 
time  were  the  independent  variables  measured  against  various  factors  obtained  from  demographics.  Briefly, 
when  individual  treatments  are  aggregated,  individuals  between  25-35  years  of  age  took  longer  to  traverse 
the  maze  when  compared  to  both  older  and  younger  groups.  Likewise,  females  and  persons  with  spatially 
oriented  academic  majors  took  longer  to  traverse  the  maze.  We  conjecture  that  these  increases  in  times  are 
due  to  well  known  gender  differences,  maturity  or  existing  training  in  spatially  oriented  tasks. 


BACKGROUND 

In  augmented  reality,  “a  participant  wears  a  see-through  display  (or  views  video  of  the  real  world  with  opaque 
HMD)  that  allows  graphics  to  be  projected  in  the  real  world”  (Barfield  &  Caudell,  2001,  pg.  6).  It  holds  great 
promise  to  enhance  human  performance  in  the  real  world  by  providing  contextually  relevant  information  to 
the  user  on  demand.  Unlike  virtual  reality  (VR),  where  the  user  is  immersed  solely  in  a  computer  generated 
environment,  augmented  reality  allows  the  user  to  see  the  real  world  with  computer  generated  images 
superimposed.  Milgram,  Takemura,  Utsumi,  and  Kishino  (1994)  use  the  terms  reproduction  fidelity,  extent  of 
presence,  and  extent  of  world  knowledge  augmented  to  describe  the  functional  characteristics  of  virtual  and 
augmented  reality  systems.  These  terms  represent  respectively  i)  the  graphics  quality,  ii)  the  degree  to  which 
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the  user  forms  an  integrated  mental  model,  and  iii)  the  amount  of  real  world  knowledge  available  via  the 
computer. 

Because  the  type  of  computer  generated  information  in  augmented  reality  is  only  generally  described,  it  is 
useful  to  further  categorize  the  computer  generated  information  into  three  classes;  information  that  is  intended 
to  be  fused  and  indistinguishable  from  the  real  world,  information  that  is  not  visible  without  augmentation 
(e.g.,  sensor  information  fused  to  a  background),  and  information  that  is  not  part  of  the  environment  but 
retains  the  characteristics  of  augmented  reality  systems,  such  as  a  head  up  display.  This  classification  scheme 
is  useful  for  two  reasons.  First,  certain  types  of  augmented  reality  technology  are  just  now  reaching  a  state  of 
maturity  where  some  human  performance  studies  may  be  conducted.  Secondly,  we  believe  that  certain  tasks 
may  be  amenable  to  particular  augmented  reality  technical  approaches  allowing  testing  to  be  conducted 
sooner  supporting  development  of  design  and  usage  guidelines.  Further,  closely  linking  research  related 
assessments  and  development  can  accelerate  augmented  reality’s  maturation  and  introduction  into  society. 

Additional  uses  of  augmented  reality  are  being  suggested  as  the  technology  matures  (see  Azuma,  1997  and 
2001).  The  literature  suggests  functional  groupings  of  augmented  reality  uses  in  manipulative  tasks,  decision 
aides,  and  navigation.  A  more  contemporary  form  of  augmented  reality  related  to  annotations  and 
visualizations  are  finding  their  way  into  modem  media  with  computer  generated  advertisements  added  to  a 
real  world  sporting  event  background.  In  military  aircraft  head  up  displays  and  helmet  mounted  sights  are 
also  examples  of  augmentation. 

One  area  of  human  performance  that  is  just  beginning  to  be  studied  in  augmented  reality  is  navigation  or 
wayfinding.  Wayfinding  is  characterized  by  acquiring  landmark,  route,  and  survey  knowledge  about  an  area 
as  described  by  Siegel  and  White  (1975).  Wayfinding  is  one  of  the  crucial  application  areas  for  augmented 
reality  in  areas  such  as  public  safety.  This  paper  explores  a  particular  type  of  navigation  we  term  search  and 
rescue.  Search  and  rescue  navigation  is  characterized  by  the  requirement  to  quickly  cover  an  entire  space 
searching  for  an  object,  finding  that  object,  and  exiting  with  it  as  quickly  as  possible.  Shortness  of  time, 
retrieval  of  the  object,  and  full  exploration  of  the  space  are  required. 

Wearable  computer  technology  makes  real-world  wayfinding  studies  feasible.  Augmented  reality  which 
employs  wearable  computers  offers  the  possibility  for  training  that  overcomes  limitations  often  experienced  in 
virtual  reality  systems,  such  as  restrictions  in  self  motion  that  can  result  in  simulator  sickness.  Additionally, 
virtual  reality  systems  require  dedicated  and  specialized  spaces  as  well  as  consume  a  relatively  large  footprint. 
However,  critical  technology  developments  remain  in  several  areas  of  augmented  reality  including  tracking, 
displays,  human  factors,  and  power.  For  example,  Azuma,  et  al.  (2001)  discuss  persistent  problems  with 
displays  such  as  the  dynamic  range  of  brightness,  limited  field  of  view,  and  fixed  contrast  ratio  that  restrict  the 
blending  of  real  and  synthetic  images.  However,  displays  for  gaming  are  making  advances  across  some  of 
these  areas  while  keeping  the  cost  per  unit  reasonable  for  consumers.  Likewise,  support  for  occlusion  in 
optical  displays  has  been  prototyped  by  including  special  LCD  panels  in  the  optical  path.  Display  sizes  for 
augmented  displays  are  also  decreasing  with  eyeglass  type  of  displays  becoming  available.  Progress  is  being 
made  across  a  broad  front  of  technology. 


EXPERIMENTAL  DESIGN 

The  experiments  described  herein  used  a  mobile  augmented  reality  system.  This  system,  called  the  Battlefield 
Augmented  Reality  System  (BARS),  was  integrated  and  configured  by  the  Naval  Research  Laboratory  for 
experimentation  and  provided  for  this  research  by  the  United  States  Army  Research  Institute  for  Behavioral 


28-2 


RTO-MP-HFM-136 


Human  Performance  Assessments 
when  Using  Augmented  Reality  for  Navigation 


Sciences.  Images  of  the  system  can  be  seen  in  Figure  1.  For  this  experiment,  the  BARS  was  configured  as  an 
augmented  reality  system  which  uses  information  that  is  not  part  of  the  environment  but  retains  the 
characteristics  of  augmented  reality  systems.  The  BARS  uses  a  Thermite  computer  from  Quantum  3D  as  the 
central  processing  unit.  The  Thermite  is  a  battery  operated  wearable  computer.  It  uses  a  1  G  Hz  Transmeta 
central  processor,  with  supporting  hardware  from  NVDIA  GeForceFX  Go  GPU  for  graphics  processing.  The 
configuration  of  the  Thermite  for  this  research  used  the  Windows  operating  system  and  provided  several 
options  for  display  outputs  (VGA  is  used  in  BARS).  Visuals  are  provided  to  the  user  with  a  MicroOptical 
SV-6  PC  Viewer.  The  SV-6  has  640  pixels  by  480  lines  resolution,  18  bit  color  depth  (262,144  colors),  an 
approximate  16  degree  x  20  degree  field  of  view,  60  Hertz  refresh,  and  adjustable  focus  from  2  to  15  feet. 
The  setting  of  focus  helps  facilitate  placing  images  at  pre-selected  distances  to  facilitate  depth  perception. 
The  maze  used  to  conduct  the  simulated  search  and  rescue  task  is  shown  in  Figure  2. 


Figure  1 :  Front  and  Side  View  of  the  Battlefield  Augmented  Reality  System  (BARS). 
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Entrance 


Figure  2:  Oblique  Photograph  of  the  Maze. 


One  hundred  thirty  six  participants,  eight  of  which  were  used  in  a  pre-testing  pilot  study,  were  employed  in 
the  study.  Seven  of  the  136  participants  were  not  used  for  procedural  reasons  and  one  for  equipment  failure 
during  the  experiment.  This  left  a  total  of  one  hundred  twenty  participants  in  the  final  sample  with  mean  age 
of  26.5  years  and  standard  deviation  of  9.9  years.  Participants  were  divided  equally  between  the  treatment 
conditions  which  were  also  balanced  for  gender. 

There  were  two  tasks  that  each  individual  participant  had  to  accomplish.  The  primary  task  was  to  completely 
traverse  the  maze  including  dead  ends  before  retrieving  the  target  object  and  then  immediately  find  the 
shortest  way  back  to  the  entrance.  The  secondary  embedded  task  was  to  answer  spatially  oriented  questions 
placed  at  five  different  stations  in  the  maze.  The  primary  purpose  of  the  secondary  task  was  to  test  spatial 
orientation  but  a  secondary  purpose  was  to  mitigate  the  occurrence  of  a  ceiling  effect  by  imposing  an 
additional  requirement  on  participant’s  working  memory.  Spatial  recall  of  the  maze  layout  and  object 
location  was  also  required. 

Six  conditions  were  evaluated.  Two  control  conditions  were  employed  without  augmentation.  In  one,  time 
started  when  the  participant  was  given  a  paper  map  to  review  at  the  entrance  of  the  maze  (figure  2).  The  map 
was  retrieved  when  the  participant  began  traversing  the  maze.  In  the  second  control  condition  the  participant 
was  not  given  a  map,  but  did  have  the  use  of  a  compass  as  they  traversed  the  maze.  Two  variables  were 
crossed  yielding  the  additional  four  experimental  conditions.  The  two  variables  were  a  continuously  on  map 
display  versus  a  user  controlled  map  (so-called  On  Demand  display)  and  a  fixed  north  up  map  (exocentric) 
versus  a  forward  up  (egocentric)  map  display.  No  maps  or  compasses  were  provided  to  participants  using 
augmented  reality  in  the  experimental  treatments. 

Prior  to  the  experiment,  participants  completed  an  informed  consent  and  demographic  questionnaire.  Parts  5 
and  6  of  the  Guilford-Zimmerman  (G-Z)  Aptitude  Survey  were  administered  for  spatial  abilities  (Consulting 
Psychologists  Press,  1976).  Participants  were  also  asked  to  answer  the  Immersive  Tendencies  Questionnaire 
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that  was  adapted  for  AR.  After  traversing  the  maze  participants  were  given  the  Presence  Questionnaire 
adapted  for  AR  and  a  spatial  recall  test  (see  Goldiez,  2004). 


RESULTS 

Data  collected  from  each  participant  consisted  of  the  time  taken  to  traverse  the  maze,  the  percent  of  the  maze 
covered,  estimations  of  Euclidian  distance  and  direction,  estimations  of  Cardinal  direction,  and  spatial  recall 
of  the  maze  layout  and  location  of  the  object.  A  variety  of  pre  and  post  hoc  questionnaires  were  also 
administered  covering  demographics,  spatial  abilities,  and  presence. 

The  results  from  the  experiment  suggest  that  a  higher  level  of  accuracy  is  achieved  using  augmented  reality. 
Less  time  was  needed  to  traverse  the  area  without  augmentation,  but  we  conjecture  that  this  was  due  in  part  to 
the  small  size  of  the  maze.  As  one  might  expect  from  the  literature,  augmented  reality  systems  employing 
egocentric  displays  result  in  better  performance  for  making  estimations  of  Euclidian  distance  and  direction 
while  exocentric  displays  result  in  better  estimation  of  Cardinal  direction.  On  Demand  displays  generally 
result  in  improved  performance  compared  to  continuously  on  displays.  See  Goldiez  (2004)  for  complete 
results. 

Additional  data  were  analyzed  regarding  the  occurrence  of  presence  and  the  impact  of  demographics  on 
performance.  Presence  is  generally  defined  as  ‘the  sense  of  being  somewhere  other  than  your  physical 
location’.  It  is  a  psychological  phenomenon  that  is  mediated  by  media  (in  the  present  context  by  the 
augmented  reality  equipment),  task,  and  predisposition  of  the  participant.  No  direct  impact  of  presence  was 
detected  using  adapted  versions  of  the  Witmer-Singer  Immersive  Tendency  and  Presence  Questionnaires 
(Goldiez,  2004)  and  the  analysis  methods  suggested  by  Witmer  and  Singer  (1998).  Demographic  information 
gathered  from  participants  was  analyzed  with  respect  to  its  impact  on  performance.  Demographic  data 
showed  no  impact  on  performance  when  individual  treatments  were  analyzed  except  for  the  well  documented 
effect  of  males  performing  better  than  females  in  spatially  oriented  tasks  were  extended  to  this  experimental 
domain. 

Further  investigation  into  the  data  revealed  some  interesting  affects  with  respect  to  presence  and 
demographics.  For  example,  a  question  added  to  the  Witmer-Singer  Immersive  Tendency  Questionnaire 
asked  participants  to  what  extent  they  rotate  maps  when  reading  them.  Those  indicating  that  they  tended  not 
to  rotate  maps  exhibited  better  performance  with  respect  to  traversal  time  than  those  indicting  that  they  tend  to 
rotate  maps  when  reading  them.  The  difference  in  traversal  times  between  these  groups  was  in  excess  of  1 00 
seconds  when  nominal  traversal  times  were  on  the  order  of  400  seconds  (p<  0.030).  It  is  important  to 
emphasize  that  this  question  was  posed  before  the  experiment  and  it  is  not  known  whether  the  individual 
actually  rotates  the  map  during  a  navigational  task.  We  infer  from  this  finding  that  individuals  who  build 
fixed  map  mental  models  do  better  in  navigation  than  those  who  do  not  build  such  models. 

Further  investigation  of  the  presence  questionnaires  was  conducted.  The  initial  approach  to  analyzing 
presence  was  to  select  three  questions  each  from  the  adapated  Immersive  Tendency  and  Presence 
Questionnaires  and  investigate  the  degree  of  statistical  correlation.  The  correlation  was  poor.  Subsequently, 
we  evaluated  the  reliability  scale  of  individual  questionnaires  using  the  clustering  suggested  in  Witmer  and 
Singer  (1998),  but  with  our  adapted  questionnaire  which  eliminated  and  added  selected  questions  (see 
Goldiez,  2004).  Reliability  as  expressed  by  Cronbach’s  alpha  was  still  below  the  .7  value  considered  to  good 
(Pallant,  2001). 
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A  complete  exploratory  factor  analysis  was  subsequently  conducted  on  the  adapted  versions  of  the  Immersive 
Tendency  and  Presence  Questionnaires.  A  completely  different  clustering  of  questions  resulted.  The  original 
work  by  Witmer  and  Singer  clustered  Immersive  Tendency  questions  using  the  terms  Focus,  Involvement,  and 
Games.  The  value  of  Cronbach’s  Alpha  for  each  factor  was  below  0.7  for  the  experiment  described  herein. 
The  new  clustering  organizes  questions  into  what  we  term  Psychomotor  Tendencies,  Cognitive  Involvement, 
and  Perceptual  Involvement.  The  regrouping  of  the  questions  under  these  latter  categories  has  Cronbach 
Alpha  values  all  above  0.7.  In  the  case  of  the  adapted  Presence  Questionnaire,  the  original  work  of  Witmer 
and  Singer  clustered  factors  into  two  categories;  Involvement/Control  and  Interface  Quality.  A  category 
called  New  Gear  was  added  to  reflect  the  wearable  augmented  reality  equipment  used  in  this  experiment 
reflecting  wearable  computing  equipment.  Reliability  scales  using  Cronbach’s  Alpha  were  all  less  than  0.7. 
New  factors  that  emerged  we  term  Human  Computer  Interface  Factors,  Technical  Factors,  and 
Kinesthetic/Tactile  exploratory  factor  analysis.  On  an  aggregate  basis,  each  of  the  new  Presence 
Questionnaire  factors  showed  improvement  compared  to  the  original  factors  although  two  factors  remain 
slightly  below  the  0.7  threshold. 

Demographics  were  also  further  analyzed  with  respect  to  the  impact  of  specific  demographic  attributes  on 
performance.  As  previously  stated,  when  individual  treatments  were  considered  the  only  demographic 
information  that  impacted  performance  when  using  augmented  reality  in  the  experimental  task  was  sex.  A 
different  approach  was  taken  to  further  analyze  demographic  data.  Ingress-Egress  Time  and  spatial  recall 
were  the  dependent  variable  aggregated  across  various  combinations  of  control  groups  and  experimental 
treatments.  Spatial  recall  was  scored  as  50  points  for  selecting  the  correct  maze  and  target  object  location,  10 
points  for  selecting  the  correct  maze  or  object,  and  0  points  for  failing  to  select  either.  The  demographic 
factors  considered  for  further  analysis  were  age,  academic  major,  profession,  and  previous  virtual  reality 
experience.  Academic  major  and  profession  were  focused  on  those  that  might  require  spatial  and/or 
quantitative  abilities.  Age  was  grouped  into  three  categories;  25  years  of  age  or  younger,  above  25  but  less 
than  or  equal  to  35,  and  above  35.  The  distribution  of  participants  by  age  was  76,  28,  and  16,  respectively. 
Academic  majors  were  classified  into  four  categories;  physical  science,  social  sciences,  humanities,  and  other. 
The  distribution  of  participants  among  these  four  categories  was  57,  24,  17,  and  22,  respectively.  Profession 
was  classified  into  three  categories;  spatial  (e.g.,  engineer),  non-spatial  (e.g.,  English  teacher),  and  other.  The 
distribution  among  these  three  categories  was  21,  20,  and  79,  respectively  (many  participants  were  students). 
38  participants  had  some  form  of  virtual  reality  experience. 

In  a  few  cases,  the  findings  from  the  additional  analysis  of  demographics  showed  some  statistically  significant 
findings.  An  ANOVA  for  traversal  time  across  all  treatments  considering  physical  science  majors  compared 
to  all  other  majors  found  that  participants  indicating  a  physical  science  major  were  statistically  different  (mean 
time  381  sec)  from  an  aggregate  of  majors  (mean  time  306  sec)  for  the  control  group  with  a  map  (F(l  ,1 8)  = 
4.22,  p  <  0.05).  Also  a  two-way  ANOVA  of  traversal  time  versus  spatial  recall  and  age  group  revealed  that 
their  interaction  is  statistically  significant  (F(4,l  1 1)  =  5.34,  p  <  0.001). 
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Mean  IE_Time 
Grouped  by  SR  and  Age  Group 


AGE  Group 

Figure  3:  Mean  IE_Time  across  Age  Group/SR. 


In  the  bar  plot  (Figure  3,  above)  of  the  mean  traversal  time  (ordinate),  it  can  be  observed  on  a  qualitative 
basis,  that  the  nature  of  the  bars  for  SR=50  (plain  bar)  is  just  opposite  than  the  two  other  bars  for  SR=  0  (cross 
hatched)  and  SR=10  (solid  filled  bar)  for  those  aged  25  to  35.  The  variability  in  traversal  for  the  age  group  ‘1’ 
(under  25  years  of  age)  is  smaller  than  the  other  age  groups.  Age  group  ‘2’  (age  25  to  35)  has  higher 
variability  than  that  of  age  group  ‘1’.  It  is  also  interesting  for  age  group  ‘2’  that  those  who  scored  perfect 
score  clocked  less  average  time  than  those  who  were  partially  able  to  or  completely  unable  to  recall  maze  and 
the  location  of  the  object.  It  is  quite  natural  to  expect  having  an  increase  in  recall  accuracy  with  an  increase  of 
time  spent  inside  the  maze,  however,  group  ‘2’  contradicts  with  this  general  belief. 


CONCLUSIONS 

This  study  has  shown  that  augmented  reality  can  improve  aspects  of  human  performance  in  search  and  rescue 
type  of  navigation  tasks.  This  study  has  also  shown  that  more  development  needs  to  occur,  particularly  in  the 
human  factors  and  assessment  aspects  of  the  technology.  Getting  firmer  grounding  in  these  two  areas  will 
provide  valuable  insights  into  where  high  payoff  technical  development  should  occur. 

With  respect  to  human  factors,  anecdotal  data  collected  points  to  the  need  for  hands  free  operation  when  using 
an  On  Demand  Display.  Likewise,  smaller  latency  between  movement  and  display  results  as  well  as  more 
stable  display  output  would  mitigate  some  orientation  issues. 

In  the  area  of  assessment,  a  larger  maze  is  desirable  to  see  where  working  memory  becomes  saturated  and 
augmentation’s  value  can  be  realized.  Also,  more  questions  that  are  applicable  across  media  and  tasks  are 
needed  to  be  added  and  vetted  for  questionnaires,  especially  with  respect  to  the  Presence  Questionnaire. 

Assessments  specifically  oriented  to  demographics  needs  to  be  conducted.  However,  some  general 
conclusions  can  be  drawn  from  the  present  work.  People  with  a  quantitative  major  or  those  in  the  middle  age 
group  (25  to  35  years  of  age)  seem  to  be  more  deliberate  and  thoughtful  in  their  use  of  augmented  reality  and 
generally  took  longer  to  traverse  the  maze.  We  believe  that  this  is  because  these  groups  might  be  more 
inquisitive  than  the  other  groups.  Also,  physical  science  majors  took  longer  to  execute  the  primary  and 
secondary  tasks  than  other  discipline  categories  in  the  control  treatment  with  a  map.  Again,  we  believe  this 
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effect  might  be  due  to  an  inquisitive  nature  of  this  type  of  individual  and  the  use  of  new  technology.  Age  also 
showed  significance  in  the  time  to  conduct  the  tasks,  but  in  two  unusual  ways.  First,  age  was  only  statistically 
significant  for  traversal  time  in  the  Exocentric  Continuous  Display  treatment  and  only  for  age  group  2  (ages 
25-35)  participants  who  exhibited  significantly  poorer  performance  than  other  age  groups  in  this  treatment. 
Within  age  group  2  (and  disregarding  treatment  type)  it  was  observed  that  those  who  had  perfect  spatial  and 
object  recall  took  less  time  to  execute  the  primary  and  secondary  tasks  than  those  who  were  at  best  partially 
correct  in  SR.  One  might  infer  from  this  data  that  physical  science  majors,  between  the  ages  of  25-35  gain 
more  by  using  augmented  reality  systems  for  search  and  rescue  navigation  because  of  the  improved 
performance  in  traversal  time  and  spatial  recall  compared  to  other  groups,  in  particular  a  control  condition 
(with  map)  where  traversal  time  suffered.  But  we  believe  that  what  these  results  really  indicate  is  the  need  for 
additional  technical  development,  human  factors  work,  and  assessment. 

The  lack  of  demographic  related  findings  across  a  broad  range  of  augmented  reality  treatments  leads  one  to 
conclude  that  further  work  in  AR  development  is  needed  in  combination  with  integrated  studies  involving 
increased  task  complexity  and  equipment  utility  so  that  issues  affecting  society’s  use  of  AR  can  be  better 
ascertained. 

REFERENCES 

Azuma,  R.  T.  (2001).  Augmented  reality:  Approaches  and  technical  challenges.  In  W.  Barfield  &  T.  Caudell 
(Eds.),  Fundamentals  of  wearable  computers  &  augmented  reality  (pp.  27-63).  Mahwah,  NJ:  Lawrence 
Erlbaum  and  Associates. 

Azuma,  R.T.,  Baillot,  Y.,  Behringer,  R.,  Feiner,  S.,  Julier,  S.,  &  MacIntyre,  B.  (2001).  Recent  advances  in 
augmented  reality.  Computer  Graphics  &  Applications,  IEEE,  21  (6),  34-47. 

Azuma,  R.  T.  (1997).  A  survey  of  augmented  reality.  Presence:  Teleoperators  &  Virtual  Environments,  6  (4), 
355-385. 

Consulting  Psychological  Press.  (1976).  Palo  Alto,  CA:  Consulting  Psychological  Press. 

W.  Barfield,  and  T.  Caudell  (2001),  “Basic  concepts  in  wearable  computers  and  augmented  reality,” 
Fundamentals  of  wearable  computers  and  augmented  reality.  W.  Barfield  and  T.  Caudell,  eds.,  Mahwah,  NJ: 
Lawrence  Erlbaum  and  Associates,  pp.  3-26,  2001. 

Goldiez,  B.  (2004).  Techniques  for  assessing  and  improving  performance  in  navigation  and  wayfinding  using 
mobile  augmented  reality.  Unpublished  doctoral  dissertation,  University  of  Central  Florida,  Orlando. 

Milgram,  P.,  Takemura,  H.,  Utsumi,  A.,  &  Kishino,  F.  (1994).  Augmented  reality:  A  class  of  displays  on  the 
reality-virtuality  continuum.  Proceeding  of  Society  for  Photographic  and  Information  Engineering, 
Telemanipulator  and  Telepresence  Technologies,  Vol.  2351,  unknown  location,  282-292. 

Pallant,  J.  (2001).  SPSS  Survival  Manual:  A  step  by  step  guide  to  data  analysis  using  SPSS.  Berkshire,  UK: 
Open  University  Press. 

Siegel,  A.  W.,  &  White,  S.  H.  (1975).  The  development  of  spatial  representations  of  large-scale  environments. 
In  H.  W.  Reese  (Ed.),  Advances  in  child  development  and  behavior  (pp.  9-55).  New  York,  NY:  Academic 
Press. 


28-8 


RTO-MP-HFM-136 


Human  Performance  Assessments 
when  Using  Augmented  Reality  for  Navigation 


Witmer,  B.  G.,  &  Singer,  M.  J.  (1998).  Measuring  presence  in  virtual  environments:  A  presence  questionnaire. 
Presence:  Teleoperators  &  Virtual  Environments,  7,  225-240. 


ACKNOWLEDGEMENT 

This  work  was  supported  in  part  by  a  grant  from  the  Office  of  Naval  Research  (N00014-03-1-0677  and  a 
Cooperative  Research  and  Development  Agreement  between  the  University  of  Central  Florida  and  the  Army 
Research  Institute  for  Behavior  Science.  The  opinions  expressed  herein,  though,  are  those  of  the  authors. 


RTO-MP-HFM-136 


28-9 


Human  Performance  Assessments 

when  Using  Augmented  Reality  for  Navigation 


28-10 


RTO-MP-HFM-136 


HUMAN  PERFORMANCE 
ASSESSMENTS  WHEN  USING 
AUGMENTED  REALITY  FOR 

NAVIGATION 

6 

University  ot 

Central 

Florida 


Brian  Goldiez,  Nabin  Saptoka,  Prashanth  Aedunuthula 

Institute  for  Simulation  &  Training 
University  of  Central  Florida 


June  2006 


Outline 


Central 

Florida 


•  Overview  of  AR 

•  Description  of  Navigation  Experiments 

•  Results  and  Analysis  of  Experimental  Data 

•  Additional  Data  Analysis  Research 

-  Presence 

-  Demographics 

•  On-Going  Research 


Augmented  Reality  Overview 


•  Overlaid  Computer  Graphics 
onto  a  Real  World  Scene- 
Contemporary 

-  Combines  real  and  virtual 

-  Interactive  in  real  time 

-  Registered  in  3D 

(Azuma,  1997,  Presence) 

•  Can  Include  Other  Sensory 
Modalities 

•  Can  Remove  as  Well  as  Add 
Content 

•  Anchoring  Determines  Virtual 
or  Augmented  Reality 


\  AR 

Type 
Task  N. 

Added 

Content 

Fused 

Multi- 

Spectral 

Self 

Motion 

Manipulation 

Cognition 
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Example  Technologies 


Photos  from  Azuma,  et.  al,  “Recent  Advances  in  Augmented  Reality 
Nov.  2001,  Computers  and  Graphics 


aat  ft 


Sample  AR  Application 


ft 


University  of 

Central 

Florida 


Navigation  Research  Using 

AR 


Central 

Florida 


AR  Supporting  Navigation 


Human  Performance  <->  Technology  &  Design  Guidelines 

Search  &  Rescue  Wayfinding 

-  No  Training  Involved 

-  Complete  Exploration  During  Ingress 

-  Expeditious  Egress 

Traverse  an  Unfamiliar  Area  (e.g.,  Maze)  to  Retrieve  an  Object 
Keep  Track  of  Relative  and  Absolute  Spatial  Orientation 

-  Dual  Task 

-  Recall  Accuracy 
Incentive  for  Performance 
Experimental  Design  (120  Participants) 

-  2x2  Between  Subjects 

-  2  Control  Conditions 

-  Secondary  Working  Memory  Task 

-  Balanced  for  Gender 
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Mobile  AR  Apparatus  UCF 
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Total  Time  In  Maze 


Control  Performs  Better  Than  Egocentric  Map 
Exocentric  (North  Up)  Map  Performs  Better  Than  Egocentric 


ANOVA 


ANACOVA  (Age  Covariate) 


1000: 


CMCNMEC  ED  X  C  XD 
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Treatment 
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CM 

EC 
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.017 
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EC 
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.094 
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.050 

Treatment 


Percent  of  Maze  Traveled 
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On  Demand  Displays  Result  in  Better  Performance 


Statistics 


Treatment 

Mean 

(%) 

Std  Deviation 

(%) 

N 

CM 

93.4 

7.4 

20 

C_NM 

96.2 

5.2 

20 

EC 

94.8 

5.4 

20 

ED 

98.2 

2.9 

20 

XC 

93.5 

11.9 

20 

XI) 

97.6 

5.1 

20 

Total 

95.6 

7.0 

120 

ANOVA 


Trtmt  (I) 

Trtmt  (J) 

A  (I-J) 
(percent) 

P- 

ED 

CM 

4.8 

.030 

XI) 

CM 

4.2 

.057 

ED 

X_C 

4.7 

.035 

XI) 

X_C 

4.0 

.067 

Ceiling  Effect  Present. 
Nevertheless  Statistics 
Relevant  for  Search 
Nav. 
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Accuracy  Scores 

(Impact  of  Small  Maze  Size) 


•  Euclidian  Directions 

-  C_M  Better  than  Exocentric 

•  X_C  (p=.043) 

•  X_D  (p=.022) 

-  E_D  >X_D  (p=  062) 

•  Euclidian  Distances 

-  C_M  Better  than  Exocentric 

•  X_C  (p=.079) 

•  X_D  (p=.023) 

-  C_M>E_D  (p=.004) 

-  E_C>E_D  (p=.013) 

-  E_C>X_D  (p=.064) 


•  Cardinal  Directions 

-  C_M  Better  than  Egocentric 

•  X_C  (p=.03l) 

•  X_D  (p=.013) 

-  E_D  >  E_C  (p=.028) 

-  X_*>E_D 

•  X_C  (p=.067) 

•  X_D  (p=.012) 

-  Gender  Effects 

•  Male>Female  (E_C) 
ANACOVA 

•  Suggestive  for  AR  T rtmts 

•  Spatial  Recall 

-  No  Statistical  Significance 
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•  Improved  Ergonomic  Design  Needed 

-  AR  Equipment  Fit  &  Ruggedness 
-Tracking,  Graphics,  &  Display  Systems 

•  Improved  Interaction  Display  Controls 

-  Hands  Free 

-  Multi-Modal 

•  Equipment  Training  or  Intuitiveness 

•  Customized  Design  Features  on  Start  Up 


Additional  Data  Analysis 

(Presence) 

Many  Widely  Used  Questionnaires 

-  SUS  (Slater-Usoh-Steed)  PQ 

-  Witmer-Singer 

•  Immersive  Tendencies  Questionnaire  and  Presence  Questionnaire 

•  Modifications  Needed  for  non-VR 

-  ITC-SOPI 

•  Cross  media 

Judging  Criteria  for  Questionnaires 

-  Reliability  (Repeatable  Results  Across  Trials  &  Participants) 

-  Validity  (Correlate  with  Subjective  Measures) 

-  Objectivity  (Shielded  from  Bias) 

-  Sensitivity  (Distinguish  Levels/Types  of  Presence) 
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Exploratory  Vs  Confirmatory  Analysis 


Confirmatory  Analysis 

Inferential  Statistics  -  Deductive 

Approach 

-  Heavy  reliance  on  probability  models 

-  Look  for  definite  answers  to  specific 
questions 

-  Emphasis  on  numerical  calculations 

-  Hypotheses  determined  at  outset 

-  Hypothesis  tests  and  formal 
confidence  interval  estimation 

Advantages 

-  Provide  precise  information  in  the 
right  circumstances 

-  Well-established  theory  and 
methods 

Disadvantages 

-  Misleading  impression  of  precision  in 
less  than  ideal  circumstances 

-  Analysis  driven  by  preconceived 
ideas 

-  Difficult  to  notice  unexpected  results 


•  Exploratory  Analysis 

•  Descriptive  Statistics  -  Inductive 
Approach 

-  Look  for  flexible  ways  to  examine  data 
without  preconceptions 

-  Attempt  to  evaluate  validity  of 
assumptions 

-  Heavy  reliance  on  graphical  displays 

-  Let  data  suggest  questions 

-  Focus  on  indications  and  approximate 
error  magnitudes 

•  Advantages 

-  Realistic  statements  of  accuracy 

-  Flexible  ways  to  generate  hypotheses 

-  Does  not  require  more  than  data  can 
support 

•  Disadvantages 

-  Usually  does  not  provide  definitive 
answers 

-  Requires  judgment 
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Subscale 

Items 

C'ronbach  Alpha  (a) 

Subscale  Items  C'ronbach  Alpha  (a) 

1 

ITQ: 

PSYCHOMOTOR 

6.  8.  10.  11.  15,  30.  35.36 

0.786 

FOCUS 

1,  3,  S.  9, 19,21,34 

0.376 

TENDENCY 

INVOLVE 

2,11,12,17,18,22,23 

0.629 

COGNITIVE 

INVOLVEMENT 

12,  17,  18.  19,  21,23 

0.733 

NEW  GEAR 

23,35,26.35,38 

0.466 

PERCEPTUAL 

INVOLVEMENT 

1,  2,  3.  9,  14,  20 

0.675 

mi 

PO: 

INVOLVE  / 
CONTROL 

1,4,10, 14,18. 19, 20, 
21,26,  28,38 

0.622 

HUMAN  COMPUTER  3,  4.  10,  14.  18.  20, 
INTERPACE  FACTORS  21,30 

0.769 

INTERFACE 

22, 23,  24 

0.361 

TECHNICAL 

19,  22,  23,  26.  28 

0.561 

QUALITY 

FACTORS 

GEAR 

31,33,36 

0.376 

KINESTHETIC 
TACTILE  FAC  TORS 

1.  13,  17 

0.500 

Table  1  Subscales  for  Goldiez  questionnaire  using  Table  2  Subscales  for  Goldiez  questionnaire  using 

Original  Witmer-Singer  clusters  (Blind  method)  Principal  component  Analysis  /Rotation  method 
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Males  Performed  Better 
Than  Females 

-  In  Both  Traversal  Time  &  % 

Covered  (Marginal  p<.10) 

-  Spatial  Visualization  &  Orientation 

(p<01) 

Other  Results  Not  Very 
Conclusive 

-  Various  Combinations  &  Groupings 
of  Other  Factors  Studied  w/o  Clear 
Outcomes 

-  Physical  Science  Majors  took  75  s. 
longer  traversal  time  than 
aggregate  of  other  majors 


•  Qualitative  Results  for 
Age  Group  2  Show  Counter 
Intuitive  Relationship  Between 
Recall  Accuracy  &  Time 


Mean  IE_Time 
Grouped  by  SR  and  Age  Group 


AGE  Group 

•  Add’l  Demographics  Study 
is  Indicated 


Current  Research  Activities 


Data  Analysis  With  Respect  to  Emergent 
Behavior 


-  Determine  Relationship  (if  any)  Between  Self  Motion 
and  Performance 

•  Suggest  In  Situ  Remediation/Training  Regime 

•  Classify  User  Types 

Additional  Presence  Research 


•  Confirm/Expand  ITQ  into  a  Questionnaire  Battery 

•  Focus  Study  on  ITQ  &  Expand  Across  Media 

-  Simulator  Hardware 

-  Robotic  Interfaces 

AR  Research  Roadmap 
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